IoT23 Data Profiling

Dataset statistics

Number of variables	18
Number of observations	565000
Missing cells	0
Missing cells (%)	0.0%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	47.4 MiB
Average record size in memory	88.0 B

Variable types

NUM	13
DATE	2
CAT	2
BOOL	1

Warnings

`resp_pkts` is highly correlated with `orig_ip_bytes` and 1 other fields	High correlation
`orig_ip_bytes` is highly correlated with `resp_pkts`	High correlation
`resp_ip_bytes` is highly correlated with `resp_pkts`	High correlation
`orig_ip_bytes` is highly skewed (γ1 = 594.8347597)	Skewed
`resp_pkts` is highly skewed (γ1 = 747.2694264)	Skewed
`resp_ip_bytes` is highly skewed (γ1 = 751.4623431)	Skewed
`ts` has unique values	Unique
`service` has 558291 (98.8%) zeros	Zeros
`orig_bytes` has 418804 (74.1%) zeros	Zeros
`resp_bytes` has 418804 (74.1%) zeros	Zeros
`conn_state` has 538319 (95.3%) zeros	Zeros
`history` has 8897 (1.6%) zeros	Zeros
`resp_pkts` has 547350 (96.9%) zeros	Zeros
`resp_ip_bytes` has 547350 (96.9%) zeros	Zeros

Reproduction

Analysis started	2022-05-31 04:07:28.735710
Analysis finished	2022-05-31 04:09:11.895353
Duration	1 minute and 43.16 seconds
Software version	pandas-profiling v2.9.0
Download configuration	config.yaml

df_index
Real number (ℝ_≥0)

Distinct	521694
Distinct (%)	92.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	440042.3587
Minimum	0
Maximum	1008747
Zeros	7
Zeros (%)	< 0.1%
Memory size	4.3 MiB

Quantile statistics

Minimum	0
5-th percentile	19448.95
Q1	138245.75
median	416827.5
Q3	712942.25
95-th percentile	949397.05
Maximum	1008747
Range	1008747
Interquartile range (IQR)	574696.5

Descriptive statistics

Standard deviation	310656.968
Coefficient of variation (CV)	0.7059706
Kurtosis	-1.29034032
Mean	440042.3587
Median Absolute Deviation (MAD)	284373.5
Skewness	0.2083309886
Sum	2.486239326e+11
Variance	9.650775174e+10
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
92	8	< 0.1%
27	8	< 0.1%
61	8	< 0.1%
80	7	< 0.1%
65	7	< 0.1%
227	7	< 0.1%
141	7	< 0.1%
64	7	< 0.1%
42	7	< 0.1%
182	7	< 0.1%
Other values (521684)	564927	> 99.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	7	< 0.1%
1	4	< 0.1%
2	4	< 0.1%
3	4	< 0.1%
4	6	< 0.1%

Value	Count	Frequency (%)
1008747	1	< 0.1%
1008745	1	< 0.1%
1008743	1	< 0.1%
1008742	1	< 0.1%
1008738	1	< 0.1%

ts
Date

UNIQUE

Distinct	565000
Distinct (%)	100.0%
Missing	0
Missing (%)	0.0%
Memory size	4.3 MiB

Minimum	2018-05-09 15:30:31.015810
Maximum	2019-07-03 14:39:13.917407

Histogram

Histogram with fixed size bins (bins=50)

id.orig_p
Real number (ℝ_≥0)

Distinct	28341
Distinct (%)	5.0%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	44494.21168
Minimum	3
Maximum	64923
Zeros	0
Zeros (%)	0.0%
Memory size	2.2 MiB

Quantile statistics

Minimum	3
5-th percentile	33704
Q1	42594
median	43763
Q3	49867
95-th percentile	58756
Maximum	64923
Range	64920
Interquartile range (IQR)	7273

Descriptive statistics

Standard deviation	10063.91686
Coefficient of variation (CV)	0.2261848559
Kurtosis	8.25123749
Mean	44494.21168
Median Absolute Deviation (MAD)	3638.5
Skewness	-2.107018995
Sum	-630574176
Variance	101282422.5
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
43763	190250	33.7%
3	7196	1.3%
123	6587	1.2%
11	1630	0.3%
5353	131	< 0.1%
68	83	< 0.1%
23	75	< 0.1%
59652	36	< 0.1%
33561	36	< 0.1%
56733	36	< 0.1%
Other values (28331)	358940	63.5%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
3	7196	1.3%
8	28	< 0.1%
11	1630	0.3%
23	75	< 0.1%
53	4	< 0.1%

Value	Count	Frequency (%)
64923	1	< 0.1%
63908	1	< 0.1%
63650	1	< 0.1%
62412	1	< 0.1%
62027	1	< 0.1%

id.resp_p
Real number (ℝ_≥0)

Distinct	62546
Distinct (%)	11.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	14840.38008
Minimum	0
Maximum	65535
Zeros	1893
Zeros (%)	0.3%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	22
Q1	23
median	7712
Q3	25022.25
95-th percentile	59353
Maximum	65535
Range	65535
Interquartile range (IQR)	24999.25

Descriptive statistics

Standard deviation	19831.86314
Coefficient of variation (CV)	1.336344692
Kurtosis	0.09543816881
Mean	14840.38008
Median Absolute Deviation (MAD)	7689
Skewness	1.23401254
Sum	-205119848
Variance	393302795.8
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
23	129873	23.0%
8080	64718	11.5%
22	62273	11.0%
2323	42197	7.5%
9527	21152	3.7%
59353	11045	2.0%
123	6696	1.2%
50	3868	0.7%
3	3463	0.6%
1	2408	0.4%
Other values (62536)	217307	38.5%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	1893	0.3%
1	2408	0.4%
2	3	< 0.1%
3	3463	0.6%
4	5	< 0.1%

Value	Count	Frequency (%)
65535	5	< 0.1%
65534	3	< 0.1%
65533	2	< 0.1%
65532	4	< 0.1%
65531	3	< 0.1%

proto
Categorical

Distinct	3
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	2.2 MiB

1	356961
2	199142
3	8897

Frequencies
Length

Value	Count	Frequency (%)
1	356961	63.2%
2	199142	35.2%
3	8897	1.6%

Frequencies of value counts

Unique

Unique	0 ?
Unique (%)	0.0%

Histogram of lengths of the category

Length

Max length	1
Median length	1
Mean length	1
Min length	1

service
Real number (ℝ_≥0)

ZEROS

Distinct	7
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.03517345133
Minimum	0
Maximum	6
Zeros	558291
Zeros (%)	98.8%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	0
median	0
Q3	0
95-th percentile	0
Maximum	6
Range	6
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	0.3752241544
Coefficient of variation (CV)	10.66782304
Kurtosis	153.0298611
Mean	0.03517345133
Median Absolute Deviation (MAD)	0
Skewness	12.13882854
Sum	19873
Variance	0.140793166
Monotocity	Not monotonic

Histogram with fixed size bins (bins=7)

Value	Count	Frequency (%)
0	558291	98.8%
5	2794	0.5%
1	2114	0.4%
2	1671	0.3%
3	83	< 0.1%
4	42	< 0.1%
6	5	< 0.1%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	558291	98.8%
1	2114	0.4%
2	1671	0.3%
3	83	< 0.1%
4	42	< 0.1%

Value	Count	Frequency (%)
6	5	< 0.1%
5	2794	0.5%
4	42	< 0.1%
3	83	< 0.1%
2	1671	0.3%

duration
Categorical

Distinct	3
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	2.2 MiB

0	418804
2	124766
1	21430

Frequencies
Length

Value	Count	Frequency (%)
0	418804	74.1%
2	124766	22.1%
1	21430	3.8%

Frequencies of value counts

Unique

Unique	0 ?
Unique (%)	0.0%

Histogram of lengths of the category

Length

Max length	1
Median length	1
Mean length	1
Min length	1

orig_bytes
Real number (ℝ_≥0)

ZEROS

Distinct	12
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.3802530973
Minimum	0
Maximum	11
Zeros	418804
Zeros (%)	74.1%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	0
median	0
Q3	1
95-th percentile	1
Maximum	11
Range	11
Interquartile range (IQR)	1

Descriptive statistics

Standard deviation	1.120707378
Coefficient of variation (CV)	2.947266927
Kurtosis	58.94515125
Mean	0.3802530973
Median Absolute Deviation (MAD)	0
Skewness	6.986809052
Sum	214843
Variance	1.255985026
Monotocity	Not monotonic

Histogram with fixed size bins (bins=12)

Value	Count	Frequency (%)
0	418804	74.1%
1	130561	23.1%
2	4856	0.9%
11	3755	0.7%
3	2758	0.5%
4	1578	0.3%
5	700	0.1%
6	528	0.1%
7	460	0.1%
8	419	0.1%
Other values (2)	581	0.1%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	418804	74.1%
1	130561	23.1%
2	4856	0.9%
3	2758	0.5%
4	1578	0.3%

Value	Count	Frequency (%)
11	3755	0.7%
10	210	< 0.1%
9	371	0.1%
8	419	0.1%
7	460	0.1%

resp_bytes
Real number (ℝ_≥0)

ZEROS

Distinct	14
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.3984884956
Minimum	0
Maximum	13
Zeros	418804
Zeros (%)	74.1%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	0
median	0
Q3	1
95-th percentile	1
Maximum	13
Range	13
Interquartile range (IQR)	1

Descriptive statistics

Standard deviation	1.33615198
Coefficient of variation (CV)	3.353050326
Kurtosis	68.74517775
Mean	0.3984884956
Median Absolute Deviation (MAD)	0
Skewness	7.827920023
Sum	225146
Variance	1.785302114
Monotocity	Not monotonic

Histogram with fixed size bins (bins=14)

Value	Count	Frequency (%)
0	418804	74.1%
1	132801	23.5%
2	4848	0.9%
13	4815	0.9%
3	1209	0.2%
4	704	0.1%
5	401	0.1%
6	363	0.1%
7	286	0.1%
8	210	< 0.1%
Other values (4)	559	0.1%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	418804	74.1%
1	132801	23.5%
2	4848	0.9%
3	1209	0.2%
4	704	0.1%

Value	Count	Frequency (%)
13	4815	0.9%
12	115	< 0.1%
11	119	< 0.1%
10	132	< 0.1%
9	193	< 0.1%

conn_state
Real number (ℝ_≥0)

ZEROS

Distinct	13
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.08575929204
Minimum	0
Maximum	12
Zeros	538319
Zeros (%)	95.3%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	0
median	0
Q3	0
95-th percentile	0
Maximum	12
Range	12
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	0.4593123568
Coefficient of variation (CV)	5.355831956
Kurtosis	96.1565304
Mean	0.08575929204
Median Absolute Deviation (MAD)	0
Skewness	8.015501744
Sum	48454
Variance	0.2109678411
Monotocity	Not monotonic

Histogram with fixed size bins (bins=13)

Value	Count	Frequency (%)
0	538319	95.3%
1	13020	2.3%
2	8913	1.6%
3	3151	0.6%
4	811	0.1%
5	448	0.1%
6	99	< 0.1%
8	73	< 0.1%
7	61	< 0.1%
9	33	< 0.1%
Other values (3)	72	< 0.1%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	538319	95.3%
1	13020	2.3%
2	8913	1.6%
3	3151	0.6%
4	811	0.1%

Value	Count	Frequency (%)
12	9	< 0.1%
11	31	< 0.1%
10	32	< 0.1%
9	33	< 0.1%
8	73	< 0.1%

history
Real number (ℝ_≥0)

ZEROS

Distinct	7
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1.423010619
Minimum	0
Maximum	6
Zeros	8897
Zeros (%)	1.6%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	1
Q1	1
median	1
Q3	2
95-th percentile	2
Maximum	6
Range	6
Interquartile range (IQR)	1

Descriptive statistics

Standard deviation	0.7132519912
Coefficient of variation (CV)	0.501227455
Kurtosis	8.322740556
Mean	1.423010619
Median Absolute Deviation (MAD)	0
Skewness	2.19857285
Sum	804001
Variance	0.5087284029
Monotocity	Not monotonic

Histogram with fixed size bins (bins=7)

Value	Count	Frequency (%)
1	345695	61.2%
2	192610	34.1%
0	8897	1.6%
5	7855	1.4%
3	6515	1.2%
4	3151	0.6%
6	277	< 0.1%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	8897	1.6%
1	345695	61.2%
2	192610	34.1%
3	6515	1.2%
4	3151	0.6%

Value	Count	Frequency (%)
6	277	< 0.1%
5	7855	1.4%
4	3151	0.6%
3	6515	1.2%
2	192610	34.1%

orig_pkts
Real number (ℝ_≥0)

Distinct	32
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1.62839292
Minimum	0
Maximum	31
Zeros	29
Zeros (%)	< 0.1%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	1
Q1	1
median	1
Q3	1
95-th percentile	3
Maximum	31
Range	31
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	1.860805603
Coefficient of variation (CV)	1.142725186
Kurtosis	103.4378683
Mean	1.62839292
Median Absolute Deviation (MAD)	0
Skewness	8.687582686
Sum	920042
Variance	3.462597493
Monotocity	Not monotonic

Histogram with fixed size bins (bins=32)

Value	Count	Frequency (%)
1	427945	75.7%
3	127746	22.6%
2	1260	0.2%
5	1114	0.2%
13	928	0.2%
15	806	0.1%
14	731	0.1%
4	718	0.1%
16	480	0.1%
8	418	0.1%
Other values (22)	2854	0.5%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	29	< 0.1%
1	427945	75.7%
2	1260	0.2%
3	127746	22.6%
4	718	0.1%

Value	Count	Frequency (%)
31	267	< 0.1%
30	65	< 0.1%
29	160	< 0.1%
28	189	< 0.1%
27	131	< 0.1%

orig_ip_bytes
Real number (ℝ_≥0)

HIGH CORRELATION
SKEWED

Distinct	1168
Distinct (%)	0.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	115.9362726
Minimum	0
Maximum	6527241
Zeros	29
Zeros (%)	< 0.1%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	40
Q1	40
median	60
Q3	76
95-th percentile	180
Maximum	6527241
Range	6527241
Interquartile range (IQR)	36

Descriptive statistics

Standard deviation	9748.689215
Coefficient of variation (CV)	84.08661931
Kurtosis	376431.1804
Mean	115.9362726
Median Absolute Deviation (MAD)	20
Skewness	594.8347597
Sum	65503994
Variance	95036941.42
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
60	221333	39.2%
40	190382	33.7%
180	126937	22.5%
76	6623	1.2%
68	4174	0.7%
56	2913	0.5%
1273	772	0.1%
57	696	0.1%
286	686	0.1%
88	602	0.1%
Other values (1158)	9882	1.7%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	29	< 0.1%
40	190382	33.7%
52	17	< 0.1%
56	2913	0.5%
57	696	0.1%

Value	Count	Frequency (%)
6527241	1	< 0.1%
3207031	1	< 0.1%
528817	1	< 0.1%
466302	1	< 0.1%
285885	1	< 0.1%

resp_pkts
Real number (ℝ_≥0)

HIGH CORRELATION
SKEWED
ZEROS

Distinct	92
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	0.7148619469
Minimum	0
Maximum	239484
Zeros	547350
Zeros (%)	96.9%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	0
median	0
Q3	0
95-th percentile	0
Maximum	239484
Range	239484
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	319.2383419
Coefficient of variation (CV)	446.5734165
Kurtosis	560526.8119
Mean	0.7148619469
Median Absolute Deviation (MAD)	0
Skewness	747.2694264
Sum	403897
Variance	101913.119
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	547350	96.9%
1	9607	1.7%
16	985	0.2%
15	832	0.1%
4	811	0.1%
5	741	0.1%
6	585	0.1%
13	400	0.1%
17	375	0.1%
3	340	0.1%
Other values (82)	2974	0.5%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	547350	96.9%
1	9607	1.7%
2	292	0.1%
3	340	0.1%
4	811	0.1%

Value	Count	Frequency (%)
239484	1	< 0.1%
7186	1	< 0.1%
7129	1	< 0.1%
6147	1	< 0.1%
5975	1	< 0.1%

resp_ip_bytes
Real number (ℝ_≥0)

HIGH CORRELATION
SKEWED
ZEROS

Distinct	1208
Distinct (%)	0.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	660.2789204
Minimum	0
Maximum	349618679
Zeros	547350
Zeros (%)	96.9%
Memory size	2.2 MiB

Quantile statistics

Minimum	0
5-th percentile	0
Q1	0
median	0
Q3	0
95-th percentile	0
Maximum	349618679
Range	349618679
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	465167.7451
Coefficient of variation (CV)	704.5018866
Kurtosis	564795.842
Mean	660.2789204
Median Absolute Deviation (MAD)	0
Skewness	751.4623431
Sum	373057590
Variance	2.163810311e+11
Monotocity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	547350	96.9%
76	4848	0.9%
40	3006	0.5%
73	1209	0.2%
216	471	0.1%
2589	313	0.1%
412	282	< 0.1%
164	260	< 0.1%
553	206	< 0.1%
3443	195	< 0.1%
Other values (1198)	6860	1.2%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	547350	96.9%
40	3006	0.5%
44	13	< 0.1%
48	3	< 0.1%
52	138	< 0.1%

Value	Count	Frequency (%)
349618679	1	< 0.1%
3835697	1	< 0.1%
2362011	1	< 0.1%
1075416	1	< 0.1%
301794	1	< 0.1%

label
Boolean

Distinct	2
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	2.2 MiB

1	333945
0	231055

Common Values
Chart

Value	Count	Frequency (%)
1	333945	59.1%
0	231055	40.9%

label2
Real number (ℝ_≥0)

Distinct	7
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	7.791566372
Minimum	0
Maximum	8
Zeros	967
Zeros (%)	0.2%
Memory size	4.3 MiB

Quantile statistics

Minimum	0
5-th percentile	7
Q1	8
median	8
Q3	8
95-th percentile	8
Maximum	8
Range	8
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	0.6943199332
Coefficient of variation (CV)	0.08911172671
Kurtosis	50.80597963
Mean	7.791566372
Median Absolute Deviation (MAD)	0
Skewness	-6.232213592
Sum	4402235
Variance	0.4820801696
Monotocity	Not monotonic

Histogram with fixed size bins (bins=7)

Value	Count	Frequency (%)
8	481057	85.1%
7	74213	13.1%
5	4915	0.9%
3	2139	0.4%
2	1587	0.3%
0	967	0.2%
1	122	< 0.1%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0	967	0.2%
1	122	< 0.1%
2	1587	0.3%
3	2139	0.4%
5	4915	0.9%

Value	Count	Frequency (%)
8	481057	85.1%
7	74213	13.1%
5	4915	0.9%
3	2139	0.4%
2	1587	0.3%

starting_point
Date

Distinct	9
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	4.3 MiB

Minimum	2018-05-21 20:52:40
Maximum	2019-12-05 15:46:36

Histogram

Histogram with fixed size bins (bins=9)

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

Cramér's V (φc)

Cramér's V is an association measure for nominal random variables. The coefficient ranges from 0 to 1, with 0 indicating independence and 1 indicating perfect association. The empirical estimators used for Cramér's V have been proved to be biased, even for large samples. We use a bias-corrected measure that has been proposed by Bergsma in 2013 that can be found here.

Count
Matrix

First rows

	df_index	ts	id.orig_p	id.resp_p	proto	duration	orig_bytes	resp_bytes	history	orig_pkts	orig_ip_bytes	label	label2	starting_point
0	756509	2018-05-13 01:13:42.015510082	42619	2323	1	0	0	0	1	1	60	1	8	2018-05-21 21:03:43
1	926165	2018-05-13 21:32:02.016247988	58178	23	1	2	1	1	1	3	180	1	8	2018-05-21 21:03:43
2	434855	2018-05-11 13:19:06.017616034	46629	61106	1	0	0	0	1	1	60	0	8	2018-05-21 21:03:43
3	1006113	2018-05-14 07:06:10.031738043	40547	2323	1	1	1	1	1	3	180	1	8	2018-05-21 21:03:43
4	757119	2018-05-13 01:17:11.007747889	43763	22653	2	0	0	0	2	1	40	0	8	2018-05-21 21:03:43
5	376930	2018-05-11 07:12:34.012142897	43763	54561	2	0	0	0	2	1	40	0	8	2018-05-21 21:03:43
6	1851	2018-05-19 19:27:25.077538013	47494	22	1	0	0	0	1	1	60	1	7	2018-05-21 20:52:40
7	800969	2018-05-13 06:32:54.034410000	53524	23	1	0	0	0	1	1	60	1	8	2018-05-21 21:03:43
8	130230	2018-05-21 01:19:15.071351051	52114	22	1	2	1	1	1	3	180	1	7	2018-05-21 20:52:40
9	980504	2018-05-14 04:02:08.021599054	40646	23	1	2	1	1	1	3	180	1	8	2018-05-21 21:03:43

Last rows

	df_index	ts	id.orig_p	id.resp_p	proto	duration	orig_bytes	resp_bytes	history	orig_pkts	orig_ip_bytes	label	label2	starting_point
564990	566350	2018-05-12 03:17:49.010447979	38570	23	1	2	1	1	1	3	180	1	8	2018-05-21 21:03:43
564991	242176	2018-05-10 16:54:33.040469885	58714	23	1	0	0	0	1	1	60	1	8	2018-05-21 21:03:43
564992	467435	2018-05-11 16:44:34.009152889	43763	5396	2	0	0	0	2	1	40	0	8	2018-05-21 21:03:43
564993	747573	2018-05-13 00:09:26.045212030	55578	23	1	0	0	0	1	1	60	1	8	2018-05-21 21:03:43
564994	192598	2018-05-10 11:40:25.013392925	43763	59450	2	0	0	0	2	1	40	0	8	2018-05-21 21:03:43
564995	278531	2018-05-10 20:46:48.008739948	52198	23	1	2	1	1	1	3	180	1	8	2018-05-21 21:03:43
564996	150990	2018-05-21 05:42:01.833146095	43575	22	1	0	0	0	1	1	60	1	7	2018-05-21 20:52:40
564997	646276	2018-05-12 12:10:10.042766094	48138	23	1	0	0	0	1	1	60	1	8	2018-05-21 21:03:43
564998	115104	2018-05-10 03:29:55.010375023	36617	23	1	2	1	1	1	3	180	1	8	2018-05-21 21:03:43
564999	880255	2018-05-13 16:02:18.020665884	54067	13757	1	2	1	1	1	3	180	0	8	2018-05-21 21:03:43

Overview

Variables

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Phik (φk)

Cramér's V (φc)

Missing values

Sample

First rows

Last rows